Pre-Training MLM Using Bert for the Albanian Language

Abstract Knowing that language is often used as a classifier of human intelligence and the development systems understand remains challenge all time (Kryeziu & Shehu, 2022). Natural Language Processing very active field study, where transformers have key role. Transformers function based on neural networks they are increasingly showing promising results. One first major contributions to transfer learning in was use pre-trained word embeddings 2010 (Joseph, Lev, Yoshua, 2010). Pre-trained models like ELMo (Matthew, et al., 2018) BERT (Delvin, 2019) trained large corpora unlabeled text result from representations has achieved good performance many underlying tasks datasets different domains. Pre-training model proven there been an improvement some aspects natural processing, paper (Dai Le, 2015). In present paper, we will pre-train task Masked Modeling (MLM) with Albanian dataset (alb_dataset) created for this purpose We compare two approaches: training using available OSCAR our alb_dataset collected. The shows discrepancies during training, especially while evaluating model.